---
title: "Workflow Overview"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Workflow Overview}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

```{r setup, message = FALSE}
library(ccesMRPprep)
library(tidyverse)
library(haven)
```

## **Step 1.** Downloading CCES data

All CCES data can be downloaded directly from dataverse, using the [`dataverse`](https://cran.r-project.org/package=dataverse) R package. The function `get_cces_dataverse()` will make this quick and simple - you only need to specify the name of the dataset, given in `?cces_dv_ids`.

```{r, eval = FALSE}
# may take about 30 seconds to download
ccc <- get_cces_dataverse("cumulative")
```

Here we will use a built-in sample of 1,000 observations for illustration (see `ccc_samp()`). In production, you will want to download all 14 datasets to your local directory via a script like [initialize-cces-downloads.R](https://github.com/kuriwaki/CCES_district-opinion/blob/master/R/initialize-cces-downloads.R) (still private).

```{r}
ccc_samp
```

## **Step 2.** Cleaning CCES data

The CCES cumulative dataset is already harmonized and cleaned, but values must be recoded so that they later match with the values of the ACS. I have created key-value pairings for the main demographic variables. The wrapper function expects a CCES cumulative dataset and recodes.

See `get_cces_question()` for how to get the *outcome* data, which is a single column from another CCES dataset. This still relies on a flat file being pre-downloaded (via `get_cces_dataverse()`).

```{r}
ccc_std <- ccc_std_demographics(ccc_samp)
count(ccc_std, age)
```

## **Step 3.** Define a Model Specification

A formula represents the variables that are in the model, how they are interacted, and which variables are random effects. We presume the formula will be used in brms. Currently we support binary outcomes. This can be set in the regression form

```{r}
fm_brm <- response ~  age + gender + educ + pct_trump + (1|cd)
```

where `response` is a binary variable or a factor/character variable that can be coerced to one using `yesno_to_binary()`.

A prior could be specified here as well, but this is not strictly necessary and can be defined when fitting the model.


## **Step 4.** Download ACS (Census) datasets for post-stratification

We provide wrappers around the great [tidycensus](https://walker-data.com/tidycensus/) package that produces ACS data and post-stratification tables from the ACS. Tailored lookup tables internally will pull out the appropriate CD-level counts and label them so that they match up with CCES keys. District-level information, which is not necessary ACS data (e.g. election outcomes) must be supplied here as well. 

```{r}
# only do this once, replacing the input with your census key
tidycensus::census_api_key(Sys.getenv("CENSUS_API_KEY"))

acs_tab <- get_acs_cces(
  varlist = acscodes_age_sex_educ,
  varlab_df = acscodes_df,
  year = 2018)

poststrat <-  get_poststrat(
  cleaned_acs = acs_tab, 
  dist_data = cd_info_2018, 
  formula = fm_brm)

 poststrat
```
You need to get a API key from the census website to run this command. See `vignette("acs")`.


## **Step 5.** (Optional) Expand Population Tables

Often, the census alone does not provide nearly all the variables one would want to post-stratify (i.e. adjust for non-representativeness). Party affiliation in Census variables is a prominent examples. The package provides open-source software to implement this expansion. See `vignette("synth")` .

## **Final Output**

Under this workflow, only three objects are needed to conduct MRP with a brms model:

1.  The model specification which can be a plain text line of a brms formula (in Step 3). All variables specified in the RHS should be in both the survey data and poststratification.
2.  The survey data
3.  The poststratification table via `get_poststrat()` and possibly expanded via synthetic methods (in Step 5).

Together, these uniquely define one MRP model for one operationalization of one question with a particular set of covariates.

